The images collected in rainy days usually have some phenomena that affect the image quality， such as the background object blocked by rain streaks and the image deformation， which have serious impact on the subsequent image analysis and application. Recently， numerous de-raining algorithms based on deep learning have been proposed and achieve good results. Most algorithms adopt supervised learning， that is， training the model on the synthetic rainy image dataset with paired labels due to the difficulty in acquiring clean background images without rain streaks from real-world rainy images. However， there are differences between synthetic and real-world rainy images on brightness， transparency， and shape of rain streaks. Thus， most de-raining algorithms based on supervised learning have poor generalization ability to real-world rainy images. Therefore， in order to improve the rain removal effect of de-raining models on the real-world rainy images， a single image de-raining algorithm based on semi-supervised learning was proposed. In the model training process of the proposed algorithm， the synthetic and real-world rainy images were added， and the difference of the first-order and second-order statistic information of feature vectors converted from the both input images were minimized to make the features of the both have same distribution. Meanwhile， in view of the complex and diverse characteristics of rain streaks， a multi-scale network was introduced to obtain richer image features and improve the performance of model. Experimental results show that， on the Rain100H dataset of synthetic rainy images， compared with Joint Deraining Network （JDNet）， Synthetic-to-Real transfer learning （Syn2Real）， the proposed algorithm improves the Peak Signal-to-Noise Ratio （PSNR） and Structural SIMilarity （SSIM） by at least 0.66 dB and 0.01 respectively. While removing rain streaks， the proposed algorithm can retain image details and color information to the greatest extent. At the same time， with the reduction of distribution discrepancy， the proposed algorithm achieves better performance on the real-world rainy images with strong generalization ability， compared with the de-raining algorithms such as JDNet and Syn2Real. The proposed algorithm is highly independent， can be applied to the existing de-raining algorithms based on supervised learning and significantly improve their de-raining effects.